Smart device and method for controlling same

ABSTRACT

The present invention provides a smart device, which is configured to track the movement of the user&#39;s head, and a method for controlling the same. The present invention provides a smart device and a method for controlling the same, the smart device comprising: a body; an output device, which can be attached to/detached from the body, and which is configured to provide multiple reference points in a peripheral space; a camera for acquiring images of the provided reference points; and a control device configured to track the movement of the smart device using the acquired images.

TECHNICAL FIELD

The present invention relates to a smart device and a method forcontrolling the same, and more particularly, to a smart device forproviding a virtual reality to a user and a method for controlling thesame.

BACKGROUND ART

Terminals may be generally classified as mobile/portable terminals orstationary terminals according to their mobility. Mobile terminals mayalso be classified as handheld terminals or vehicle mounted terminalsaccording to whether or not a user can directly carry the terminal.

Functions of mobile terminals are diversified. These functions include,for example, data and voice communication, photographing and videoshooting through a camera, voice recording, music file playback througha speaker system, and outputting an image or video to a display unit.Some terminals are equipped with an electronic gaming function ormultimedia player function. In particular, recently released mobileterminals are capable of receiving multicast signals for providingvisual content, such as broadcast and video or television programs. Inorder to perform such functions, the mobile terminals may be basicallyconnected to other devices or networks using various communicationprotocols, and provide ubiquitous computing to the user. That is, mobileterminals have evolved into smart devices that enable connectivity to anetwork and ubiquitous computing.

In recent years, as smart devices are increasingly used to play gamesand watch images, there are attempts to implement virtual reality onsmart devices in order to provide users with a better experience.Virtual reality is based on stereoscopic images provided to users. Mostsmart devices are basically not designed to realize stereoscopic images.Therefore, in order to view stereoscopic images, separate software orhardware such as a dedicated application for realizing a stereoscopicimage is required. In addition, even if a smart device is capable ofrealizing a complete stereoscopic image without any separate software orhardware, a separate extension device is required in order to allow auser to sufficiently enjoy stereoscopic images using a display of thesmart device, which is relatively small. Recently, auxiliary devicescapable of sufficiently providing a user with virtual reality, that is,a stereoscopic image enlarged to a sufficient size by using the displayof a smart device have been developed according to the need. Inparticular, considering user convenience, these auxiliary devices aremostly designed to be mounted directly on the user's head, and the smartdevice may be mounted in the auxiliary devices to substantially providevirtual reality.

Unlike general virtual-reality-based functions such as watching moviesor normal application execution, some functions, such as gaming may beperformed as intended only when dynamic motion of a user is tracked. Inorder to optimally provide virtual reality regarding such functions, thehead motion of the user reflecting change in the user's gaze needs to betracked. Thus, since the smart device is worn on the user's head by theauxiliary device, motion of the head may be tracked by tracking themotion of the smart device. However, while smart devices are capable ofproviding most of the functionality required for virtual reality, theymay not completely track their own motion (i.e., head motion) due to thelimited functionality of the sensor.

Therefore, along with the above-mentioned auxiliary device, the smartdevice needs to be further improved so as to actually track motionthereof, that is, the head motion of the user, in order to optimallyperform the virtual-reality-based function. In addition, a method foroptimally controlling the improved smart device is required in order toperform the function of tracking the head motion, and further to improvethe related user experience.

DISCLOSURE Technical Problem

The present invention is directed to solving the above-mentionedproblems and other problems. Accordingly, an object of the presentinvention is to provide a smart device configured to track motion of auser's head.

Another object of the present invention is to provide a method foroptimally tracking motion of a user's head using the smart device.

Technical Solution

The object of the present invention can be achieved by providing a smartdevice including a body, an output device detachable from the body andconfigured to provide a plurality of reference points in a surroundingspace, a camera configured to acquire images of the provided referencepoints, and a control device configured to track motion of the smartdevice using the acquired images.

The output device and the camera may continuously provide referencepoints and acquire images thereof while the smart device is moving. Thereference points may have predetermined patterns to be easilyrecognized, the patterns being different from each other. The outputdevice may provide reference points formed by infrared light, and thecamera may include an infrared camera capable of photographing theinfrared reference points.

The control device may construct a three-dimensional map ofpredetermined reference points based on positions of a predeterminednumber of the predetermined reference points acquired from the images,continuously detect a relative positional change of the predeterminedreference points with respect to the three-dimensional map using theimages, and continuously detect a position of the smart device based onthe detected positional change. In addition, the control device mayextend the three-dimensional map based on positions of additionalreference points as a range of motion of the smart device extends.

The output device may include a light source configured to emit infraredlaser, and a converter configured to convert the emitted infrared laserinto the plurality of reference points. The converter may include adiffusing member configured to uniformly disperse the emitted infraredlaser. In this case, the converter may further include a filterconfigured to selectively pass the infrared laser dispersed by thediffusing member to emit the reference points. The converter may beconfigured to diffract the emitted infrared laser to form the pluralityof reference points. The converter may be configured to reflect theemitted infrared laser in a plurality of directions to form theplurality of reference points.

The output device may be disposed at a location spaced apart from thebody to provide the plurality of reference points. The output device mayfurther include a fin configured to dissipate heat and to contractivelyextend from the output device, wherein the output device may beconfigured to adjust an orientation thereof to provide the referencepoints in a desired space. The smart device may share a plurality ofreference points provided by an output device of another smart device totrack the motion thereof. The smart device may further include a sensingdevice configured to sense attachment and detachment of the outputdevice.

The smart device may further include a head-mounted device configured toaccommodate the body, the head-mounted device being worn on a head of auser. The controller may track motions of the head-mounted device andthe head of the user wearing the smart device by tracking the motion ofthe smart device.

In another aspect of the present invention, provided herein is a methodfor controlling a smart device, including searching for a plurality ofreference points in a surrounding space when it is necessary to trackmotion of a smart device, emitting the reference points and searchingfor an output device available in the smart device when the plurality ofreference points is not found, and operating the available output devicewhen the output device is found.

The operating may include removing an output device mounted on the smartdevice from the smart device, and disposing the removed output device ata position spaced apart the smart device. The operating may includeoperating one of the output device of the smart device and an outputdevice of another smart device.

The method may further include notifying a user that the output deviceshould be removed from the smart device so as to be used at a remotelocation, the notifying being performed prior to the operating step. Themethod may further include sensing whether the output device has beenremoved from the smart device, the sensing being performed prior to thenotifying.

The method may further include notifying a user that the availableoutput device should be searched for when the available output device isnot found.

The searching may be performed when the smart device executes a specificapplication or when motion of the smart device occurs beyond apredetermined range.

The method may further include, when the plurality of reference pointsis found in the searching, continuously acquiring, by the smart device,images of the reference points while the smart device is moving,constructing a three-dimensional map of predetermined reference pointsbased on positions of a predetermined number of the predeterminedreference points acquired from the images, continuously detecting arelative positional change of the predetermined reference points withrespect to the constructed three-dimensional map using the images, andcontinuously detecting a position of the smart device based on thedetected positional change.

The searching of the output device may include searching for an outputdevice mounted on the smart device and an output device removed from thesmart device, or searching for an output device of the smart device andan available output device of another smart device.

In another aspect of the present invention, provided herein is a smartdevice including a body, an output device detachable from the body andconfigured to provide a plurality of reference points in a surroundingspace, a camera configured to acquire images of the provided referencepoints, and a control device configured to track motion of the smartdevice using the acquired images, wherein, when the plurality ofreference points is not found through the camera, the control device maynotify a user that the output device should be removed from the smartdevice before the output device is operated.

The control device may be configured to sense whether the output devicehas been removed from the smart device before notifying that the outputdevice should be removed. The control device may immediately operate theoutput device when it is sensed that the output device has been removed,wherein, when it is sensed that the output device is mounted, thecontrol device may notify the user that the output device should beremoved.

The control device may be configured to share reference points providedby an output device of another smart device to track the motion of thesmart device.

When the plurality of reference points is not found, the control devicemay be configured to operate one of the output device of the smartdevice and an output device of another smart device.

When there is no available output device, the control device may notifythe user that an available output device should be searched for.

Advantageous Effects

In the present invention, a smart device has an output device configuredto provide reference points in a surrounding space. Relative change inposition of the smart device according to motion of the smart device maybe detected using the reference points, and motion of a user's head maybe tracked based on change in position of the smart device. Therefore,the smart device according to the present invention may provide betteroptimized virtual reality to the user.

In addition, in the control method according to the present invention, atime of a motion may be determined by detecting a special case in whichmotion tracking is required. In addition, in the control method, thesmart device and the output device thereof may be appropriately preparedto track motion after tracking is determined. Therefore, the controlmethod of the present invention may effectively track the motion of thesmart device and the head, and allow the user to more convenientlyexperience virtual reality.

The scope of applicability of the present invention will become apparentfrom the following detailed description. It should be understood,however, that the detailed description and specific examples, such asthe preferred embodiments of the invention, are given by way ofillustration only, since various changes and modifications within thespirit and scope of the invention will be apparent to those skilled inthe art.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of a smartdevice described in this application.

FIGS. 2 and 3 are perspective views showing an example of a smart deviceviewed from different sides.

FIG. 4 is an exploded perspective view showing a head-mounted device,which is an accessory used together with a smart device.

FIG. 5 is a perspective view showing the head-mounted device with acover removed;

FIG. 6 is a perspective view illustrating a smart device worn by a userusing a head-mounted device and an output device.

FIG. 7 is a conceptual diagram illustrating reference points emittedfrom an output device.

FIG. 8 is a flowchart illustrating a method for tracking motions of asmart device and a user's head.

FIG. 9 is a perspective view showing an output device mounted on a smartdevice.

FIG. 10 is a perspective view showing the structure of the outputdevice.

FIG. 11 is a perspective view showing extendable legs or fins of theoutput device.

FIG. 12 is a perspective view showing the output device mounted on aseparate cradle.

FIG. 13 is a perspective view showing a smart device worn by a userusing a head-mounted device and a modified output device.

FIG. 14 is a perspective view showing the modified output device.

FIG. 15 is a cross-sectional view showing the modified output device.

FIG. 16 is a perspective view showing the modified output devicedisposed on the floor;

FIG. 17 is a perspective view illustrating a smart device worn by a userusing a head-mounted device and a modified output device.

FIG. 18 is a perspective view showing the modified output device.

FIG. 19 is a perspective view showing in detail the conversion structureof the modified output device.

FIG. 20 is a perspective view showing the modified output devicedisposed on the floor;

FIG. 21 is a flowchart illustrating an example of a method forcontrolling a smart device described in the present application.

FIG. 22 is a conceptual diagram showing examples of notificationsprovided to the user in the method for controlling a smart device.

BEST MODE

Description will now be given in detail according to exemplaryembodiments disclosed herein, with reference to the accompanyingdrawings. For the sake of brief description with reference to thedrawings, the same or equivalent components may be provided with thesame reference numbers, and description thereof will not be repeated. Ingeneral, a term such as “module” and “unit” may be used to refer toelements or components. Use of such a term herein is merely intended tofacilitate description of the specification, and the term itself is notintended to give any special meaning or function. In the presentdisclosure, that which is well-known to one of ordinary skill in therelevant art has generally been omitted for the sake of brevity. Theaccompanying drawings are used to help easily understand varioustechnical features and it should be understood that the embodimentspresented herein are not limited by the accompanying drawings. As such,the present disclosure should be construed to extend to any alterations,equivalents and substitutes in addition to those which are particularlyset out in the accompanying drawings.

It will be understood that although the terms first (1st), second (2nd),etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are generally onlyused to distinguish one element from another.

It will be understood that when an element is referred to as being“connected with” another element, the element can be directly connectedwith the other element or intervening elements may also be present. Incontrast, when an element is referred to as being “directly connectedwith” another element, there are no intervening elements present.

A singular representation may include a plural representation unless itrepresents a definitely different meaning from the context.

Terms such as “comprise”, “include” or “have” are used herein and shouldbe understood that they are intended to indicate an existence of severalcomponents, functions or steps, disclosed in the specification, and itis also understood that greater or fewer components, functions, or stepsmay likewise be utilized. Moreover, due to the same reasons, it is alsounderstood that the present application includes a combination offeatures, numerals, steps, operations, components, parts and the likepartially omitted from the related or involved features, numerals,steps, operations, components and parts described using theaforementioned terms unless deviating from the intentions of thedisclosed original invention.

Smart devices presented herein may be implemented using a variety ofdifferent types of terminals. Examples of such terminals includecellular phones, smart phones, laptop computers, digital broadcastterminals, personal digital assistants (PDAs), portable multimediaplayers (PMPs), navigators, slate PCs, tablet PCs, ultrabooks, wearabledevices (for example, smart watches, smart glasses, head mounteddisplays (HMDs)), and the like. By way of non-limiting example only,further description will be made with reference to particular types ofsmart devices. However, such teachings apply equally to other types ofsmart devices, such as those types noted above.

FIG. 1 is a block diagram to describe a smart device related to thepresent application. A general configuration of the smart device isdescribed with reference to FIG. 1 as follows.

First of all, the smart device 100 may include components such as awireless communication unit 110, an input unit 120, a sensing unit 140,an output unit 150, an interface unit 160, a memory 170, a controller180, a power supply unit 190, and the like. It is appreciated thatimplementing all of the components shown in FIG. 1 is not a requirement,and that greater or fewer components may alternatively be implemented.Moreover, the real shapes and structures of the aforementionedcomponents are not illustrated all but the shapes and structures of somesignificant components are shown in the drawings following FIG. 1. Yet,it is apparent to those skilled in the art that components describedwithout being illustrated can be included in the smart watch to embodythe functions of a smart device.

In particular, among the above-listed components, the wirelesscommunication unit 110 typically includes one or more modules whichpermit communications such as wireless communications between the smartdevice 100 and a wireless communication system, communications betweenthe smart device 100 and another smart device, communications betweenthe smart device 100 and an external server. Further, the wirelesscommunication unit 110 typically includes one or more modules whichconnect the smart device 100 to one or more networks.

To facilitate such communications, the wireless communication unit 110may include one or more of a broadcast receiving module 111, a mobilecommunication module 112, a wireless Internet module 113, a short-rangecommunication module 114, and a location information module 115.

The input unit 120 includes a camera 121 (or an image input unit) for animage or video signal input, a microphone 122 (or an audio input unit)for an audio signal input, and a user input unit 123 (e.g., a touch key,a push key, etc.) for receiving an input of information from a user.Audio or image data collected by the input unit 123 may be analyzed andprocessed into user's control command.

The sensing unit 140 is typically implemented using one or more sensorsconfigured to sense internal information of the smart device, thesurrounding environment of the smart device, user information, and thelike. For example, the sensing unit 140 may include a proximity sensor141 and an illumination sensor 142. If desired, the sensing unit 140 mayalternatively or additionally include other types of sensors or devices,such as a touch sensor, an acceleration sensor, a magnetic sensor, agravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGBsensor, an infrared (IR) sensor, a finger scan sensor, a ultrasonicsensor, an optical sensor (for example, the camera 12 a), the microphone12 b, a battery gauge, an environment sensor (for example, a barometer,a hygrometer, a thermometer, a radiation detection sensor, a thermalsensor, and a gas sensor, among others), and a chemical sensor (forexample, an electronic nose, a health care sensor, a biometric sensor,and the like), to name a few. The smart device 100 disclosed in thepresent specification may be configured to utilize information obtainedfrom the sensing unit 140, and in particular, information obtained fromone or more sensors of the sensing unit 140, and combinations thereof.

The output unit 150 is typically configured to output various types ofinformation, such as audio, video, tactile output, and the like. Theoutput unit 150 may include a display unit 151, an audio output unit152, a haptic module 153, and an optical output module 154. The displayunit 151 may have an inter-layered structure or an integrated structurewith a touch sensor in order to facilitate a touchscreen. Thetouchscreen may provide an output interface between the smart device 100and a user, as well as function as the user input unit 123 whichprovides an input interface between the smart device 100 and the user.

The interface unit 160 serves as an interface with various types ofexternal devices that can be coupled to the smart device 100. Theinterface unit 160, for example, may include any of wired or wirelessports, external power supply ports, wired or wireless data ports, memorycard ports, ports for connecting a device having an identificationmodule, audio input/output (I/O) ports, video I/O ports, earphone ports,and the like. In some cases, the smart watch 100 may perform assortedcontrol functions associated with a connected external device, inresponse to the external device being connected to the interface unit160.

The memory 170 is typically implemented to store data to support variousfunctions or features of the smart device 100. For instance, the memory170 may be configured to store application programs (or applications)run in the smart watch 100, data or instructions for operations of thesmart device 100, and the like. Some of these application programs maybe downloaded from an external server via wireless communication. Otherapplication programs may be installed on the smart device 100 at time ofmanufacturing or shipping, which is typically the case for basicfunctions of the smart device 100 (for example, receiving a call,placing a call, receiving a message, sending a message, and the like).It is common for application programs to be stored in the memory 170,installed on the smart device 100, and launched by the controller 180 toperform operations (or functions) for the smart device 100.

The controller 180 typically functions to control overall operations ofthe smart device 100, in addition to the operations associated with theapplication programs. The controller 180 may provide or processinformation or functions appropriate for a user by processing signals,data, information and the like, which are inputted or outputted by thevarious components depicted in the above description, or runningapplication programs stored in the memory 170.

Moreover, in order to launch an application program stored in the memory170, the controller 180 can control at least one portion of thecomponents described with reference to FIG. 1. Furthermore, thecontroller 180 controls at least two of the components included in thesmart device 100 to be activated in combination to launch theapplication program.

The power supply unit 190 can be configured to receive external power orprovide internal power in order to supply appropriate power required foroperating elements and components included in the smart device 100. Thepower supply unit 190 may include a battery. In particular, the batterymay include at least one of a built-in battery or a replaceable (ordetachable) battery.

At least some of the above components may operate in cooperation witheach other to implement operation, control, or a control method of asmart device according to various embodiments described below. Also, theoperation, control, or control method of the smart device may beimplemented on a smart device by executing at least one applicationprogram stored in the memory 170.

In the accompanying drawings, the smart device 100 is illustrated ashaving a body in the form of a bar. However, examples described in thepresent application are not limited thereto, and the smart device mayhave various structures and shapes. That is, the configuration anddescription of a specific type of the smart device 100 may be applied toother types of smart devices as well as to the specific type of thesmart watch 100.

Following the general configuration of the smart device 100 describedabove, the structure of the smart device 100 is described with referenceto relevant drawings. In this regard, FIGS. 2 and 3 are perspectiveviews showing an example of a smart device viewed from different sides.More specifically, FIG. 2 is a perspective view showing the frontportion of the smart device 100, and FIG. 3 is a perspective viewshowing the rear portion of the smart device 100. FIGS. 2 and 3 show theoverall structure of the smart device 100, and therefore alldescriptions are based on FIGS. 2 and 3, except when particularlyreferenced figures are referred to.

As described above, the smart device 100 has a body having a bar shapeas a whole. The shape of the body may be varied as needed. Here, thebody may be understood as a concept referring to the smart device 100taken as at least one assembly.

The smart device 100 includes a case (e.g., a frame, a housing, a cover,etc.) that defines an appearance. As shown in the figures, the smartdevice 100 may include a front case 101 and a rear case 102. Variouselectronic components may be disposed in the internal space formed bycoupling of the front case 101 and the rear case 102. The coupled cases101 and 102 may also form the body, or the contour thereof. At least onemiddle case may be additionally disposed between the front case 101 andthe rear case 102.

A display unit 151 may be disposed on the front surface of the smartdevice 100 to output information. As shown in the figures, the displayunit 151 may be exposed to the outside of the front case 101 and thusmay form the front surface of the smart device 100 together with thefront case 101.

In some cases, electronic components may be mounted on the rear case 102as well. The electronic components that may be mounted on the rear case102 may include a detachable battery, an identification module, and amemory card. In this case, a rear cover 103 for covering the mountedelectronic components may be detachably coupled to the rear case 102.Therefore, when the rear cover 103 is detached from the rear case 102,the electronic components mounted on the rear case 102 are exposed tothe outside so as to be accessible.

As shown in the figures, when the rear cover 103 is coupled to the rearcase 102, a part of the lateral surface of the rear case 102 may beexposed. In some cases, the rear case 102 may be completely covered bythe rear cover 103 when coupled to the rear cover 103. The rear cover103 may be provided with an opening for exposing a camera 121 b and asound output unit 152 b to the outside.

These cases 101, 102 and 103 may be formed by injection molding ofsynthetic resin or may be formed of metal such as, for example,stainless steel (STS), aluminum (Al), titanium (Ti) or the like.

The smart device 100 may be configured such that one case provides theinternal space, unlike the example above where a plurality of casesprovides the internal space for accommodating various electroniccomponents. In this case, a unibody smart device 100 in which syntheticresin or metal extends from the lateral surface to the rear surface maybe implemented.

The smart device 100 may include a waterproof part (not shown) forpreventing water from infiltrating the body of the smart device 100. Forexample, the waterproof part may include a waterproof member providedbetween the window 151 a and the front case 101, between the front case101 and the rear case 102, or between the rear case 102 and the rearcover 103 to seal the internal space when the window, the cases and thecover are coupled.

The smart device 100 may include a display unit 151, first and secondsound output units 152 a and 152 b, a proximity sensor 141, anillumination sensor 142, a light output unit 154, first and secondcameras 121 a and 121 b, first and second manipulation units 123 a and123 b, a microphone 122, and an interface unit 160.

Hereinafter, a description will be given of the smart device 100 inwhich the display unit 151, the first sound output unit 152 a, theproximity sensor 141, the illumination sensor 142, the light output unit154, the first camera 121 a and the first manipulation unit 123 a aredisposed on the front surface of the body, the second manipulation unit123 b, the microphone 122, and the interface unit 160 are disposed onthe lateral surface of the body, and the second sound output unit 152 b,and the second camera 121 b are disposed on the rear surface of thebody, as shown in FIGS. 2 and 3.

However, these elements are not limited to the aforementionedarrangement. These elements may be omitted or replaced with other onesas needed, or disposed on other surfaces. For example, the firstmanipulation unit 123 a may not be provided on the front surface of theterminal body, and the second sound output unit 152 b may be provided onthe lateral surface of the terminal body rather than on the rear surfaceof the terminal body.

The display unit 151 displays (outputs) information processed in thesmart device 100. For example, the display unit 151 may displayexecution screen information about an application program driven by thesmart device 100, or user interface (UI) and graphic user interface(GUI) information according to the execution screen information.Further, as already mentioned above, when the smart device 100 is wornon the user's head using the auxiliary device, the display portion 151may provide the user with a stereoscopic image for virtual reality.

The display unit 151 may include at least one of a liquid crystaldisplay (LCD), a thin film transistor-liquid crystal display (TFT-LCD),an organic light-emitting diode (OLED) display, a flexible display, a 3Ddisplay, or an electronic ink (e-ink) display. The display unit 151 mayinclude a display module (not shown) and a window 151 a covering thedisplay module. The display module may include a display element such asan LCD or an OLED as described above, and may be a component thatactually displays image information. The window 151 a may be disposed ata portion of the display module that is exposed to the user, and mayprotect the display module from the outside. In addition to thisprotection function, the window 151 a should allow the informationdisplayed on the display module to be seen by the user therethrough.Accordingly, the window 151 a may be formed of a material havingappropriate strength and transparency. The display module may bedirectly attached to the rear surface of the window 151 a. The displaymodule may be directly attached to the window 151 in various ways, andan adhesive may be most conveniently used for direct attachment.

In addition, two or more display units 151 may be provided depending onimplementation of the smart device 100. In this case, the plurality ofdisplay units may be spaced apart from one another or integrallyarranged on one surface of the smart device 100, or may be disposed ondifferent surfaces of the smart device 100.

The display unit 151 may include a touch sensor (not shown) configuredto sense a touch on the display unit 151 such that a control command maybe received in a touch manner. The touch sensor may use at least one ofvarious touch techniques such as a resistive touch technique, acapacitive touch technique, an infrared touch technique, an ultrasonictouch technique, and a magnetic field touch technique. As one example,as in the resistive touch technique and the capacitive touch technique,the touch sensor may be configured to convert a change in pressureapplied to a specific portion of a touchscreen or a change incapacitance occurring at a specific portion of the touchscreen into anelectrical input signal. The display unit 151 may configure atouchscreen as a kind of a touch input device in a smart device togetherwith the touch sensor. The display unit 151 may display predeterminedimage information while operating as a touchscreen, which is a userinterface. That is, the display unit 151 may function not only as theoutput unit 150 but also as the input unit 120. When the display unit151 is touched, the touch sensor may sense the touch, and the controlunit 180 may generate a control command corresponding to the touch basedon the sensing. The content input in a touch manner may be a characteror a number, or a menu item that may be indicated or designated invarious modes.

The touch sensor may be configured in the form of a film having a touchpattern and disposed between the window 151 a and the display module onthe rear surface of the window 151 a, or may be a metal wire directlypatterned on the rear surface of the window 151 a. Alternatively, thetouch sensor may be integrated with the display module. For example, thetouch sensor may be disposed on the substrate of the display module, ormay be provided inside the display module.

In this way, the display unit 151 may form a touchscreen together withthe touch sensor. In this case, the touchscreen may function as the userinput unit 123 (see FIG. 1). If needed, a physical key (e.g., a pushkey) may be additionally arranged adjacent to the display unit 151,which may be a touchscreen, as the user input unit 123 for convenientinput by the user.

The first sound output unit 152 a may be implemented as a receiver fortransmitting a call sound to a user's ear and the second sound outputunit 152 b may be implemented as a loud speaker for outputting variousalarm sounds or multimedia playback sound.

The window 151 a of the display unit 151 may be provided with a soundhole for emitting a sound generated from the first sound output unit 152a. However, the present invention is not limited thereto. The sound maybe configured to be emitted along an assembly gap between the structures(for example, a gap between the window 151 a and the front case 101). Inthis case, the appearance of the smart device 100 may become simplerbecause a hole independently formed for sound output is invisible orhidden from the outside.

The light output unit 154 is configured to output light for notifying auser of the occurrence of an event. Examples of the events may includemessage reception, call signal reception, a missed call, an alarm, aschedule notification, e-mail reception, and reception of informationthrough an application. The control unit 180 may control the lightoutput unit 154 to terminate light output when the event confirmation ofthe user is sensed.

The first camera 121 a processes an image frame of a still image or amoving image obtained by the image sensor in the photographing mode orthe video call mode. The processed image frame may be displayed on thedisplay unit 151 and may be stored in the memory 170.

The first and second manipulation units 123 a and 123 b may becollectively referred to as a manipulating portion as an example of theuser input portion 123 manipulated to receive a command for controllingthe operation of the smart device 100. The first and second manipulationunits 123 a and 123 b may be manipulated in any manner (tactile manner),such as touch, push, and scroll that provide a tactile sensation to theuser. The first and second manipulation units 123 a and 123 b may bemanipulated in a manner such as a proximity touch and a hovering touchthat does not provide a tactile sensation to the user.

While the first manipulation unit 123 a is illustrated as a touch key inthe figures, the present invention is not limited thereto. For example,the first manipulation unit 123 a may be configured as a push key(mechanical key) or a combination of a touch key and a push key.

Various contents may be set to be input through the first and secondmanipulation units 123 a and 123 b. For example, the first manipulationunit 123 a may receive input of commands for a menu, a home key, cancel,a search, and the like, and the second manipulation unit 123 b mayreceive inputs of commands such as adjustment of volume of sound outputfrom the first or second sound output unit 152 a or 152 b and switch ofthe display unit 151 to a touch recognition mode.

The manipulation units 123 a and 123 b may include a touch input devicehaving a structure similar to that of the touchscreen applied to thedisplay unit 151 described above. Unlike the touchscreen, themanipulation units 123 a and 123 b may be configured to simply allowcommands to be input therethrough without displaying image information,and the touch input device applied to the manipulation units may becalled a touch pad.

A rear input unit (not shown) may be provided on the rear surface of theterminal body as another example of the user input unit 123. The rearinput unit is manipulated to receive a command for controlling theoperation of the smart device 100, and various input content may be setto be input. For example, commands such as power on/off, start, end andscrolling, adjustment of the volume of sound output from the first andsecond sound output units 152 a and 152 b, switch of the display unit151 to the touch recognition mode and the like may be input through therear input unit. The rear input unit may be implemented to allow inputsuch as touch input, push input, or a combination thereof.

The rear input unit may be disposed to overlap the front display unit151 in the thickness direction of the terminal body. For example, therear input unit may be disposed at the upper end portion of the rearsurface of the terminal body such that the user may easily manipulatethe rear input unit when the user holds the terminal body with one hand.However, the present invention is not limited thereto, and the positionof the rear input unit may be changed.

When the rear input unit is provided on the rear side of the terminalbody as described above, a new type of user interface using the rearinput unit may be realized. If at least a part of the functions of thefirst manipulation unit 123 a provided on the front surface of theterminal body is replaced by the touchscreen or the rear input unit, andthus the first manipulation unit 123 a is not disposed on the frontsurface of the terminal body, the display unit 151 may be configuredwith a larger screen.

The smart device 100 may include a fingerprint recognition sensor forrecognizing a fingerprint of the user, and the controller 180 may usefingerprint information sensed through the fingerprint recognitionsensor as an authentication means. The fingerprint recognition sensormay be embedded in the display unit 151 or the user input unit 123.

The microphone 122 may be configured to receive the user's voice andother sounds. The microphone 122 may be provided at a plurality ofplaces to receive stereophonic sound.

The interface unit 160 serves as a path through which the smart device100 is connected to an external device. For example, the interface unit160 may be at least one of a connection terminal (e.g., a USB port) forconnection with another device (e.g., an earphone, an external speaker),a port for short-range communication (e.g., an IrDA port, a Bluetoothport, a wireless LAN port, etc.), or a power supply terminal forsupplying power to the smart device 100. The interface unit 160 may beimplemented in the form of a socket for accommodating an external cardsuch as a SIM (Subscriber Identification Module) card, a UIM (UserIdentity Module) card or a memory card for storing information.

A second camera 121 b may be disposed on the rear surface of theterminal body. In this case, the second camera 121 b has a photographingdirection substantially opposite to that of the first camera 121 a.

The second camera 121 b may include a plurality of lenses arranged alongat least one line. The plurality of lenses may be arranged in a matrix.This camera may be called an array camera. When the second camera 121 bis configured as an array camera, images may be captured in various waysusing a plurality of lenses, and a better quality image may be obtained.

The flash 124 may be disposed adjacent to the second camera 121 b. Theflash 124 shines light toward a subject when the subject is photographedwith the second camera 121 b.

The second sound output unit 152 b may be additionally disposed on theterminal body. The second sound output unit 152 b may implement a stereosound function together with the first sound output unit 152 a and maybe used to implement a speakerphone mode during a call.

The terminal body may be provided with at least one antenna for wirelesscommunication. The antenna may be embedded in the terminal body orformed in the case. For example, an antenna constituting a part of thebroadcast reception module 111 (see FIG. 1A) may be configured to bedrawn out from the terminal body. Alternatively, the antenna may beformed as a film type and attached to the inner surface of the rearcover 103, or a case including a conductive material may be configuredto function as an antenna.

The terminal body may be provided with a power supply 190 (see FIG. 1A)for supplying power to the smart device 100. The power supply 190 mayinclude a battery 191 built in the terminal body or detachably providedto the exterior of the terminal body.

The battery 191 may be configured to receive power through a power cableconnected to the interface unit 160. In addition, the battery 191 may beconfigured to be wirelessly charged through a wireless charger. Wirelesscharging may be implemented by magnetic induction or resonance (magneticresonance).

In the figures, the rear cover 103 may be coupled to the rear case 102so as to cover the battery 191 to restrict release of the battery 191and to protect the battery 191 from external impact and foreignsubstances. When the battery 191 is detachably attached to the terminalbody, the rear cover 103 may be detachably coupled to the rear case 102.

Although not shown in FIGS. 2 and 3, the substrate may be installed inthe cases 101 and 102, and various electronic components, especiallyvarious processors constituting the control unit 180, may be mounted onthe substrate together with other circuits and devices assisting thesame. The elements 110 to 190 shown in FIG. 1 may be directly installedon the substrate so as to be controllable by the control unit 180, ormay be installed on or in the cases 101 and 102 and electricallyconnected to the substrate. For example, although exposed to theoutside, the window 151 (i.e., the touch sensor) and the display modulemay each be connected to the substrate through wiring. Accordingly, thecontrol unit 180 may be referred to by various names such as acontroller and a control device, and may control all the elements of thesmart device 100. Such controlled elements include not only the elementsincluded in FIG. 1 but also other elements described below. Accordingly,the control unit 180 may be a practical element that appropriatelyperforms a control method of the present application, which will bedescribed later, by controlling the operations of the other elements.For this reason, all the detailed steps performed in the control methodmay be features of the control unit 180.

The smart device 100 may be provided with accessories to protect theexterior thereof or to supplement or extend the functionality of thesmart device 100. One example of such accessories may be a cover orpouch that covers or accommodates at least one surface of the smartdevice 100. The cover or pouch may be operatively connected with thedisplay unit 151 to extend the functionality of the smart device 100.Another example of the accessories may be a touch pen for supplementingor extending touch input to the touchscreen.

The smart device 100 may use an accessory as described above to providethe user with a stereoscopic image for virtual reality. The accessorymay be worn on the head for convenience of the user, and the smartdevice 100 may be detachably mounted in the accessory. That is, theaccessory may function substantially as an adapter for holding the smartdevice 100 on the user's head to provide virtual reality. Inconsideration of the function of holding the smart device 100 on thehead, the accessory will be described as a head-mounted device in thefollowing with reference to the related drawings. Since the head-mounteddevice 1 is integrated with the smart device 100 in terms of bothstructure and functionality, it may be substantially considered a partof the smart device 100 and not a separate device. FIG. 4 is an explodedperspective view showing a head-mounted device, which is an accessoryused together with a smart device, and FIG. 5 is a perspective viewshowing the head-mounted device with a cover removed. For simplicity,the head-mounted device 1 will be referred to as a mounted device in thefollowing description. FIGS. 6, 13 and 17 show the smart device 100 thatis actually worn on the user's head along with the mounted device 1, andaccordingly will be referred to along with FIGS. 4 and 5 in thefollowing description of the mounted device 1.

Referring to FIG. 1, the mounted device 1 may have a body 10. Basically,the body 10 may have a hollow shape in order to allow the user to viewstereoscopic images provided by the smart device 100 therethrough. Thatis, the body 10 may have open front and rear portions, and thus may forma hollow tube 11 as a whole. The body 10 may include a partition 12 fordividing the internal space of the body into left and right spaces. Thepartition 12 may separate the left and right fields of view of the userfrom each other by dividing the space inside the body 10. Therefore, theuser may see the provided stereoscopic image more clearly. The partition12 may also serve to reinforce the hollow body 10.

The mounted device 1 may be arranged around the user's eyes and broughtinto close contact with the user's face, in order to make the userconcentrate on the provided stereoscopic images and to minimizeinfiltration of light interfering with viewing of the images into thedevice during use of the device. However, the nose protruding from theuser's face may block the device from closely contacting the face. Themounted device 1, more precisely the body 10, may be configured toaccommodate the user's nose. Thus, the body 10 may have a first seat 12a formed in the partition 12 to accommodate the user's nose. The body 10may also have a second seat 12 b formed in a lower portion thereof toaccommodate the user's nose. The first and second seats 12 a and 12 bmay be connected to each other and form a continuous seat foraccommodating the user's nose. When the mounted device 1 is worn by theuser using the first and second seats 12 a and 12 b, the user's nose isstably accommodated. Thus, the mounted device 1 may be brought intoclose contact with the user. Accordingly, the user may comfortably wearthe device 1 and clearer images may be provided to the user.

The mounted device 1 may have a cover 20. The cover 20 may basicallyinclude a body 21. The body 21 may be formed of a plate-shaped memberfor supporting the rear surface of the smart device 100, as shown inFIG. 5, so as to stably support the smart device 100. The cover 20 mayalso have a flange 22 extending from the body 21. The flange 22 maybasically assist the body 21 to stably support the smart device 100. Theflange 22 may extend from the upper, lower, left and right sides of thebody 21 and be arranged with the body 21 to allow the cover 20 to have apredetermined space for accommodating the smart device 100. The smartdevice 100 may be stably accommodated in the space of the cover 20 suchthat the rear surface of the smart device 100 is supported by the body21 and the display 151 on which the stereoscopic image is displayedfaces the user.

Since the smart device 100 displays stereoscopic images while beingaccommodated in the cover 20, the smart device 100, particularly thedisplay unit 151 thereof, needs to be correctly aligned with the fieldof view of the user in order to provide a correct stereoscopic image tothe user. Alignment may be obtained when the center of the smart device100 is placed at the center of the cover 20, or more accurately, thecenter of the body 21. As such, the cover 20 may have a rib 23, as shownin FIGS. 4 and 5, to align the smart device 100 with the center of thecover 20. The rib 23 may be provided to the edge of the cover 20, morespecifically, the body 21, and extend from the edge toward the inside ofthe cover 20 or the body 21. The rib 23 guides insertion of the smartdevice 100 into the cover 10 on both sides. Thus, when the smart device100 is placed on the body 21, the smart device 100 may be positioned atthe center of the cover 10 to be aligned with the user's line of sight.

As shown in FIG. 5, the cover 20 may include at least one opening 21 ato 21 c allowing operation of the various devices of the smart device100 when the smart device is mounted in the device 1, that is, while thesmart device is accommodated in the cover 20. Even when the smart device100 is mounted in the mounted device 1, the smart device 100 needs tooperate various devices thereof in relation to stereoscopic images orother functions. For example, such devices may include input devicessuch as a button, a switch and a microphone, output devices such as aspeaker, a camera flash and an earphone jack, and a sensing device forsensing information about various surrounding environments. Consideringthe structure of the smart device 100 shown in FIGS. 2 and 3, theopenings 21 a of the body 21 may be provided for the second sound outputunit 152 b, the second camera 121 b, and the flash 124, and the openings21 b and 21 c of the flange 22 may be provided for the interface 160 andthe earphone jack. The openings 21 a to 21 c may be formed at pointswhere the corresponding devices are placed when the devices areaccommodated in the cover 20. The user may operate the smart device 100using the input devices through the openings 21 a to 21 c, the outputdevices may output predetermined information, and the sensing device maycollect necessary information. Therefore, the mounted device 1 may allowthe smart device 100 to perform other functions while providingstereoscopic images.

When the smart device 100, which is an independent device, is not usedin the mounted device 1, the user should be allowed to use the smartdevice 100 to perform other functions. Thus, the smart device 100 shouldbe able to be detached from the mounted device 1 whenever the userdesires. For this reason, the element for mounting the smart device 100may be configured to be detachable from the mounted device 1. That is,the cover 20 may be detachably provided on the body 10, specifically,the front portion thereof. To allow such detachment, various mechanismsmay be used, for example, a projection and a groove to be coupled toeach other may be employed.

Referring back to FIG. 4, the mounted device 1 may have goggles 30. Likeother elements 10 and 20, the goggles 30 may include a body 31. Thegoggles 30 may include an optical device to deliver a stereoscopic imagedisplayed on the smart device 100 to the user in a more appropriatestate. More specifically, the goggles 30 may include openings 31 aformed in the body 31. The openings 31 a correspond to the left andright eyes of the user, respectively. The goggles 30 may also includeleft and right lenses 35 provided in the openings 31 a, respectively. Inaddition, a gasket 35 a may be provided between the lenses 35 and theopenings 31 a. The gasket 35 a may catch the lens 35 to prevent the lens35 from being separated from the body 31 and prevent light from leakingthrough a clearance between the lenses 35 and the openings 31 a in orderto allow the user to view the images with better quality. Due to suchinstallation arrangement, when the user wears the device to viewstereoscopic images, the lenses 35 are respectively placed on the user'seyes. Accordingly, as a kind of optical device, the lenses 35 mayenlarge the stereoscopic image displayed on the display 151 of the smartdevice 100, and deliver the enlarged image to the user's eyeballs,thereby providing a better experience for the user in viewing thestereoscopic image. Further, if the smart device 100 provides astereoscopic image requiring separate device such as polarizing glasses,the lenses 35 may be configured to perform the same function as that ofthe polarizing glasses as well as the enlarging function.

The goggles 30 may have a flange 32 extending from the body 31. Theflange 32 may be a member that directly contacts the face of the user.Thus, the flange 32 may have a shape that matches the contour of theuser's face to block light from infiltrating into the goggles to reachthe eyes of the user. In addition, the mounted device 1 may be stablysupported against the user's face by the flange 32. In addition, thenose protruding from the user's face may obstruct the device 1 fromclosely contacting the face when the user wears the mounted device 1. Inaddition to the body 10 described above, a device, more specifically,the goggles 30, may also be configured to accommodate the user's nose.More specifically, as shown in FIG. 5, the goggles 30 may have a firstseat 31 a formed in the body 31 to accommodate the user's nose. Theseats 12 a, 12 b and 31 a of the body 10 and the goggles 30 may beconnected to each other to form a seat configured to accommodate thenose. Thus, the mounted device 1 may be brought into close contact withthe user to provide an optimal stereoscopic image.

The goggles 30 may also include a holder 34 formed on the flange 32 tomount a strap or band (not shown). The holder 34 is formed as an openingand is provided on both sides of the flange 32. Although not shown, thestrap may be fitted into the holder 34, and may be adjusted to fit thesize of the user's head. Thus, using the strap, the device may be wornso as not to be separated from the head of the user. The strap may befabricated to have a predetermined elasticity. In this case, the elasticstrap may allow the device to be worn more stably on the user's head andto more closely contact the user's face.

The mounted device 1 described above is a kind of accessory usedtogether with the smart device 100 and may basically allow the smartdevice 100 to be worn on the user's head. Further, the mounted device 1may expand the stereoscopic image displayed on the smart device 1 andprovide the expanded image to the user. Thus, the smart device 100 maybe used together with the mounted device 1 to provide the user withenhanced virtual reality.

A typical function based on virtual reality, for example, movie watchingor a typical application, may be appropriately performed by providing astereoscopic image alone. However, some functions, such as gaming,additionally use active interaction between the provided stereoscopicimages and the user, and accordingly continuous measurement of theuser's motion, i.e., tracking of motion, may be required. In particular,the head motion of the user reflecting change in the user's gaze amongthe user's motions needs to be tracked in order to optimally providevirtual reality. Since the smart device 100 is worn on the user's headby the mounted device 1 and moved along with the head as describedabove, motion of the head may be tracked easily and accurately bytracking the motion of the smart device 1.

Although the smart device 100 may provide most of the functionalityrequired for virtual reality, it does not track its own motioncompletely due to the limited functionality of the sensor. Morespecifically, the smart device 100 has a gyro sensor or an accelerationsensor capable of measuring motion or pose of the smart device withinthree-dimensional space. These sensors are capable of sensingtranslation within a limited range and rotation, but may notcontinuously sense translation beyond the range. On the other hand, themotion of the user's head usually involves continuous translationalmotion. Accordingly, the smart device 100 may not accurately track itsmotion and the motion of the user's head associated therewith by its ownsensor alone, and thus may additionally include a separate mechanism fortracking.

FIG. 6 is a perspective view illustrating a smart device worn by a userusing a head-mounted device and output device. FIG. 7 is a conceptualdiagram illustrating reference points emitted from an output device.FIG. 8 is a flowchart illustrating a method for tracking motions of asmart device and a user's head. With reference to these figures, thetracking mechanism of the smart device 100 will be described in detailbelow. Since the smart device 100 is mounted using the mounted device 1,the smart device 100 and the user's head are moved together.Accordingly, the tracking mechanism may track the motion of the user'shead by tracking the motion of the smart device 100, as previouslydescribed. Thus, tracking of motion of the smart device 100 by thetracking mechanism will be described below, and unless otherwisespecifically stated, disclosure of tracking of motion of the device 100corresponds to and should be regarded as disclosure of tracking themotion of the user's head. Further, the mounted device 1 is notdetachably installed on the detachable smart device 100. Instead, themounted device 1 may have a module that is integrated therewith andprovides the same function as that of the device 100. In this case, thedevice may be an independent device specifically for virtual reality.For this independent virtual reality device, examples of all thetracking mechanisms described above and described below may be appliedwhile maintaining the same technical concept without specialmodification.

First, as a tracking mechanism, the smart device 100 may include anoutput device 200, as shown in FIG. 6. The output device 200 may beconfigured to generate a plurality of reference points R. Further, asshown in FIG. 7, the output device 200 may be configured to provide thegenerated reference points R in a surrounding space thereof. Morespecifically, the reference points R provided in the surrounding spacemay be formed by predetermined light, and thus may travel a considerablylong distance. Thus, as shown in FIG. 7, the reference points R may beprojected onto the wall W and/or the ceiling C of the indoor space.Since the reference points R are actually emitted from the output device200 to travel a long distance, the output device 200 may also be calledan emitter.

The reference points R may utilize light from various light sources.Among these light sources, infrared light, particularly an infraredlaser, has high coherency, and thus may allow the reference points R tobe projected onto the wall W or the ceiling C without being distortedwhile traveling a long distance. In addition, since infrared light isinvisible, the projected reference points R may not degrade theappearance of the indoor space or interfere with other activities in theindoor space. For these reasons, the reference points R may be formed byinfrared light. Depending on such reference points R, the smart device100 may measure change in position thereof and track the motion thereofbased on such measurement, as described later. Therefore, consideringthis function, the reference points R may be called featured points orlandmarks. Further, in order to correctly track the motion, thereference points R need to be easily recognized by the smart device 100,and thus may have a predetermined shape or pattern, as shown in FIG. 7.Furthermore, it may be desirable for the reference points R to bedistinguished from each other in order to track change in position andtrack motion. Thus, the reference points R (or at least a part thereof)may have different patterns or shapes as shown in FIG. 7. The detailedstructure and operation for emission of reference points will bedescribed in more detail later.

Referring back to FIG. 6, the output device 200 may be detachablymounted on the smart device 100 or the body thereof. More specifically,the output device 200, i.e., the terminal thereof, may be inserted intothe interface unit 160 of the smart device 100, which is installed inthe mounted device 1 through the opening 21 b of the cover 20. Throughconnection of the terminal and the interface unit 160, the output device200 may be supplied with power and exchange data with the body of thesmart device 100, specifically, with the control unit 180. If the outputdevice 200 includes a separate battery, the battery may be charged bypower supplied from the smart device 100 or the body thereof.

In addition, the output device 200 may emit reference points R from thesmart device 100 or the mounted device 1, as shown in FIG. 7. That is,the output device 200 may emit reference points R while mounted on thesmart device 100 or the mounted device 1, as also shown in FIG. 6. Inthis case, since the smart device 100 and the mounted device 1 are wornon the user's head, the output device 200 may also be described asemitting the reference points R directly from the user's head. However,as will be described later, the reference points R serve to provide asort of reference for the moving smart device 100, that is, aninvariable background, and accordingly the positions of the referencepoints R are preferably unchanged in order to detect and track change inposition of the smart device 100. That is, the reference points Rpreferably remain fixed or stationary. If the output device 200 providesthe reference points R while being mounted on the smart device 100, itmay be difficult to provide fixed or stationary reference points R sinceit will move along with the smart device 100. Thus, the output device200 first needs to be removed or detached from the smart device 100 soas not to move along with the smart device 100. For the same reason, theseparated output device 200 must be stably positioned so as not to movein order to provide stationary reference points R. Thus, in all of theexamples described below in this application, it is assumed that theoutput device 200 is operated when it is separated from the smart device100 and placed at a remote location. For this reason, referring again toFIG. 7, the output device 200 may emit reference points R at a locationremote from the smart device 100 or the mounted device 1. That is, theoutput device 200 may first be detached from the smart device 100 or themounted device 1 and then disposed at a remote location appropriate tooutput the reference points R in the indoor space. For example, theoutput device 200 may be disposed at the center of the indoor space toemit uniform reference points R, or may be disposed at a higherposition, such as a table. In addition, the separated output device 200must take a proper pose to uniformly emit the reference points R in thespace. Thus, the output device 200 may have a support for supporting thebody thereof in a proper position. The output device 200 may be mountedon a separate cradle so as to be supported in a proper position. Theoutput device 200 may be supplied with powered by the cradle duringoperation.

In the present application, the smart device 100 may have modifiedoutput devices 200 and 300 in addition to the output device 100described above. Since the features of the output device 100 describedabove are general and common, the other output devices 200 and 300 maybasically have the same features.

In addition, as a tracking mechanism, the smart device 100 may include acamera 121 b, as shown in FIG. 6. The camera 121 b may acquire theimages of the provided reference points R. Since the smart device 100 ispositioned in front of the user's eyes to provide a stereoscopic image,the line of sight of the camera 121 b may be somewhat aligned with theline of sight of the eyes. Thus, the image acquired by the camera 121 bmay substantially correspond to the image viewed by the user. For thisreason, the positional change of the reference points that may bedetected in the obtained image may directly reflect the motion of theuser's head. More specifically, since the front surface of the smartdevice 100, that is, the display unit 151, is oriented toward the user'seyes to provide images, the rear surface of the smart device 100 facingaway from the display unit 151 may be relatively exposed to the outside.Therefore, the camera 121 b disposed on the rear surface may be exposedto the outside through the opening 21 a of the cover 20 to captureimages of the reference points R. However, depending on the orientationof the smart device 100, the camera exposed to capture the images may bechanged.

When the reference points R are formed by visible light, the camera 121b may be used without any special modification. However, if thereference points R are formed by infrared light, the camera 121 b may beconfigured to photograph infrared reference points. For example, thecamera 121 b may further include a filter installed in front of the lensof the camera 121 b to pass only infrared light. That is, by attachingsuch a filter to the front of the camera 121 b, the images of theinfrared reference points R may be captured without any additionalstructural change. The smart device 100 may further include a dedicatedinfrared camera 121 c in the body thereof, as shown in FIG. 3. Theinfrared camera 121 c may be embedded in the mounted device 1 or may bemounted on the smart device 100 using a separate accessory.

The output device 200 and the camera 121 b may continuously providereference points and continuously acquire images thereof while the smartdevice 100 is moving. Based on the acquired images, a change in positionof the reference points and corresponding motion of the device 100 maybe tracked by the smart device 100, more specifically, the control unit180 thereof. Hereinafter, a method for tracking the motion using thereference points and the images thereof will be described in more detailwith reference to FIG. 8. Since the control unit 180 substantiallycontrols all the steps of FIG. 8 as mentioned above, the features of allthe steps described below may be features of the control unit 180. Inaddition, as described above, the smart device 100 may have modifiedoutput devices 300 and 400 as well as the output device 200. The devices300 and 400 may be may be applied to the tracking method described belowin the same manner as the output device 200.

Referring to FIG. 8, the smart device 100 may first provide a pluralityof reference points R in a surrounding space (S1). These referencepoints R may be actually emitted from the output device 200 of the smartdevice 100 and be projected onto the wall W or the ceiling C of thesurrounding space. The reference points R may have specific patterns orshapes to be easily recognized by the smart device 100 or controller 180in the steps after the providing step S1. In addition, at least some ofthe reference points R may have different shapes or patterns such thatthey may be distinguished from each other. Furthermore, the outputdevice 200 may emit the reference points R from the smart device 100 orthe mounted device 1. Preferably, however, the output device 200 mayemit the reference points R at locations remote from the smart device100 or the mounted device 1, as described in detail above. Since thereference points R provide a reference for motion tracking of the smartdevice 100 in the following steps, the providing step S1 may becontinuously performed while the motion is generated and tracking isperformed. That is, the same reference points R may be continuouslyemitted in the surrounding space while the head motion is generated andtracking thereof is performed.

After the providing step S1, the smart device 100 may acquire images ofthe reference points R (S2). In the acquiring step S2, the images may beactually captured by the camera 121 b. As described above, since theline of sight and the viewing angle of the camera 121 b substantiallymatch the line of sight and viewing angle of the user, the imagesobtained by the camera 121 b may be substantially identical to theimages viewed by the user. Due to this identity, the motion of the smartdevice 100 may be matched to the motion of the user's head. Further, inorder to track the motion of the smart device 100 in subsequent steps,change in position of the smart device 100 must be continuouslydetected. To this end, images reflecting the current position of thesmart device 100 may be continuously required. Accordingly, theacquiring step S2 may be continuously performed while the motion isgenerated and tracked. That is, different images may be continuouslyacquired while the head motion occurs and tracking is performed.

The smart device 100 may then construct a three-dimensional map or athree-dimensional structure of predetermined reference points from theacquired images (S3). The three-dimensional map or structure of thereference points obtained in the constructing step S3 may be used as akind of background or reference for determining change in position ofthe reference points R. However, it may be inefficient to obtain athree-dimensional map or structure for all the reference points.Therefore, as shown in FIG. 7, a three-dimensional map or structure fora predetermined number of reference points R1 (for example, five points)may be obtained. Substantially, in the constructing step, the smartdevice 100 may select a limited number of images, i.e., key frames, fromamong a plurality of acquired images in consideration of calculationefficiency. The positions of the reference points R1-R5 are calculatedfrom the key frames using a method such as triangulation. Finally, athree-dimensional map or structure may be constructed using thisposition information.

Once the three-dimensional map or structure is constructed, the smartdevice 100 may continuously detect change in positions of thepredetermined number of reference points with respect to thethree-dimensional map or structure (S4). As described above, since theline of sight of the camera 121 b is somewhat coincident with the lineof sight of the user's eyes, the relative positions of the referencepoints R in an image obtained by the camera 121 b are substantiallyidentical to the relative positions actually shown to the user.Therefore, positional change of the predetermined reference points R1 toR5 may have a direct correlation with positional change of the user'shead as well as positional change of the smart device 100 having thecamera 121 b. Therefore, in order to detect change in position of thesmart device 100, change in positions of a predetermined number ofreference points R1 to R5 may be detected. More specifically, acquiredimages may be compared with a three-dimensional map or structure that iscontinuously constructed. Through this comparison, whether the positionsof the reference points R1 to R5 in the image have changed with respectto the reference points R1 to R5 in the three-dimensional map orstructure may be detected. Such comparison and detection may becontinuously performed, and positional change of the reference points R1to R5 may be continuously obtained.

Therefore, based on the detected positional change, the smart device maycontinuously detect the position thereof (S5). More specifically, basedon the positional change of the reference points R1 to R5 detected bydirect correlation between the reference points and the position of thesmart device 100, change in position of the smart device 100 may bedetected. For the same reason, the detected positional change of thesmart device 100 may represent a relative position of the smart device100 with respect to a specific start position (i.e., a position withinthe three-dimensional map). Accordingly, the position of the smartdevice 100 may be detected in each image based on the positional changeof the detected reference points R1 to R5. Further, in accordance withthe correlation described above, the detected position of the smartdevice 100 may be matched directly to the position of the user's head.

As shown in FIG. 8, while the motion of the smart device 100 occurs andtracking is performed, the providing step S1 may continuously providethe same reference points R, but steps S2 to S5 may be continuouslyrepeated to calculate the changing position information for motiontracking. Thus, the positions of the smart device 100 and the user'shead may be continuously detected, and the smart device 100 and theuser's head may be tracked based on these positions. In the iterativeloop of steps S2 to S5, unlike the other steps, the constructing step S3may not be repeated each time an image is acquired (S2) since it is astep of forming a background. However, if motion of the smart device 100occurs beyond the range of the previously constructed three-dimensionalmap or structure, the acquired image and the three-dimensional map maynot be properly compared with each other. Accordingly, in this case, thethree-dimensional map or structure may be extended (S3). Morespecifically, new or additional reference points R1 to R5 may beselected from the acquired image according to the extended range ofmotion of the smart device 100 and a three-dimensional map or structuremay be constructed through the same steps as described above.

The sensors in the smart device 100, for example, gyro sensors oracceleration sensors, cannot continuously detect and track the motion,in particular, the translational motion, of the smart device 100.However, these sensors may detect whether or not the translationalmotion has occurred by simply sensing the translational motion within alimited range. Accordingly, when the smart device 100 detects occurrenceof the motion, in particular, the translation motion using thesesensors, it may perform the tracking procedure from the providing stepS1 to the other steps S2-S5. In addition, since the sensors are capableof detecting the motion, particularly the translational motion, withinthe limited range, the above-described tracking procedure may be startedwhen the motion, that is, the translational motion, occurs beyond thepredetermined range. If the positional change of the reference points R1to R5 is not detected for a predetermined time in steps S4 and S5 duringthe tracking procedure, the tracking procedure may be automaticallyterminated.

Following this tracking method, the detailed structure and operation ofthe output device 200 will be described below for better understandingof the examples of the present application. In this regard, FIG. 9 is aperspective view showing an output device mounted on a smart device, andFIG. 10 is a perspective view showing the structure of the outputdevice. FIG. 11 is a perspective view showing extendable legs or fins ofthe output device, and FIG. 12 is a perspective view showing the outputdevice mounted on a separate cradle.

In terms of structure, the output device 200 may include a body 210 anda head 220, which is coupled to the body 210. The body 210 may beconnected to the smart device 100 or the body thereof. For thisconnection, the output device 200, i.e., the body 210, may have aterminal 211, which may be inserted into the interface unit 160 of thesmart device 100. The head 220 is configured to emit a plurality ofreference points R. Thus, the head 220 and the body 210 may form aninternal space for accommodating various components that create thereference points R.

In terms of functionality for generating a plurality of reference pointsR, the output device 200 may include a light source 231 and a converter.The light source 231 may be configured to emit an infrared laser, inconsideration of coherency and invisibility as mentioned above. As thelight source 231, an infrared laser diode may be employed as arepresentative example among various components which may be employed.The converter may be configured to convert infrared laser emitted fromthe light source 231 into the plurality of reference points R. Theconverter may include a diffusing member 232 configured to uniformlydisperse the emitted infrared laser, as shown in the figures. Forexample, the diffusing member 232 may be formed of an opaque memberwhich is advantageous in dispersing the laser. In addition, the outputdevice 200, more specifically, the converter, may include a filter 233that selectively passes the infrared laser dispersed by the diffusionmember 232. As shown, the head 220 may be formed to have an internalspace sufficient to allow the infrared laser having passed through thediffusing member 232 to be more uniformly dispersed, and may selectivelypass the dispersed laser. Therefore, the head 220 may be formed of adome-shaped member and may function as the filter 233 at the same time.The filter 233 (i.e., the head 220) may have a plurality of openings,holes, or slits 221 for selective passing the laser. The openings 221may have a predetermined shape or pattern that is easily identified bythe smart device 100, and the laser passing through the openings 221 mayhave this shape and pattern. Further, different patterns may be providedsuch that the generated reference points R are distinguished from eachother. Furthermore, the openings 221 may be formed throughout the filter233 for uniform emission of the reference points R.

More specifically, the light source 231 may be disposed below thediffusing member 232, and the filter 233 may be disposed above thediffusion member 232 with sufficient spacing from the diffusing member231. The diffusing member 232 may have a portion for laser emissionwider than a portion onto which the laser is incident so as tofacilitate dispersion. For example, the diffusing member 232 may be aninverted conical member, as shown in the figures. The infrared laseremitted from the light source 231 may be dispersed while passing throughthe diffusing member 232, and the laser having passed through thediffusing member 232 may be more uniformly dispersed while travelingfrom the diffusing member 232 to the filter 233. The filter 233 may passthe dispersed laser only through the openings 221, and thus the laserthat is dispersed while passing through the openings 221 may beconverted into a plurality of reference points R. In addition, the laserdispersed through the filter 233 may have a predetermined patternaccording to the pattern of the openings 221. Further, since the filter233 is a dome-shaped member having uniformly dispersed openings 221, thereference points R may be emitted throughout 180 degrees, and thereference points R may be uniformly distributed in the surroundingspace, as shown in FIG. 9.

As clearly shown in FIG. 12, the output device 200 may include asubstrate 234 on which various electronic components for controlling theoperation of the output device are mounted. The output device 200 mayalso have a switch 236 installed on the substrate 234. The output device200 may be operated or stopped by manipulating the switch 236. Theoutput device 200 may be operated or stopped using the terminal 211. Forexample, the smart device 100 may generate a signal for controlling theoperation of the output device 200, which may be transmitted to theoutput device 200 via the terminal 211 to control, i.e., operate orstop, the output device 200. In addition, since the terminal 211 iselectrically connected to the smart device 100, i.e., the interface unit160, connection and disconnection between the terminal 211 and the smartdevice 100 may trigger different electrical signals. Accordingly, theoperation and stop of the output device 200 may be controlled usingsignals generated at the time of connection and disconnection. Forexample, when the output device 200 that is turned on is mounted on thesmart device 100, the output device 200 may be immediately turned off.On the other hand, when the output device 200 in the off state mountedon the smart device 100 is removed from the smart device 100, the outputdevice 200 may be turned on immediately so as to emit the referencepoints R at the remote locations. The output device 200 may include abattery 235 for supplying power to other components. The battery 235 maybe charged by the smart device 100 or other power sources via theterminal 211. Thus, the output device 200 may operate using the battery235 even when removed from the smart device 100. The components 234 to236 described above may all be accommodated in the body 210 and the head220.

The output device 200 may generate a lot of heat since many electroniccomponents are integrated within a small area. Accordingly, the outputdevice 200 may further include fins 212 as a heat dissipating structure.As shown in FIG. 11, the fins 212 may be configured to contractivelyextend from the output device 200. Accordingly, the output device 200may have a compact appearance while having a heat dissipating structure.More specifically, the output device 200 may include a recess 213 formedin the body 210, and the fins 212 may be accommodated in the recess 213.The output device 200 may also include a groove 214 formed along thesidewall of the recess 213, and the hinge shaft 212 a of the fin 212(see FIG. 12) may be inserted into the groove. Thus, the fin 212 may bemovably installed in the output device 200 due to movement of the hingeshaft 212 a along the groove 214. This coupling structure may enable thefin 212 to move within the recess 213 when the hinge shaft 212 a movesalong the groove 214, as shown in FIG. 11(b). As the groove 214 extendsdownward, the fin 212 may also move in the same direction and thusprotrude from the output device 200 as shown in the figure. Thus, theprotruded fins 212 may support the output device 200, and accordinglythe output device 200 may operate stably at a location remote from thesmart device 100 without a separate cradle. Since the protruded fins 212substantially increase the surface area of the output device 200, theoutput device 200 may be cooled. Further, to boost the cooling effect,the protruded fins 212 may be further extended from the output device200, as shown in FIG. 11(c). Actually, the fins 212 may be radiallyexpended from the output device 200, i.e., the body 210. Accordingly,since the heat dissipation area may be further increased and moresmoothly contact ambient air, the fins 212 may cool the output device200 more effectively. When unfolded, the fins 212 do not interfere withother components proximate to the bottom of the output device 200.Accordingly, the output device 200 may be mounted on the cradle 240 withthe fins 212 unfolded. The modified output devices 300 and 400, whichwill be described later, may also have such fins 212, and thedescription given above and the related drawings may be equally appliedto the modified output devices 300 and 400.

The smart device 100 may further include modified output devices 300 and400 in addition to the output device 200. These devices 300 and 400 maybe distinguished from the output device 200 in terms of the method ofconverting the infrared laser into reference points R. The output device300 between these modified output devices will be described in detailbelow with reference to the related drawings. FIG. 13 is a perspectiveview showing a smart device worn by a user using a head-mounted deviceand a modified output device, and FIG. 14 is a perspective view showingthe modified output device. FIG. 15 is a cross-sectional view showingthe modified output device, and FIG. 16 is a perspective view showingthe modified output device disposed on the floor.

In terms of structure, the output device 300 may include a body 310 anda head 320 coupled to the body 310, as in the case of the output device200. The body 310 may be connected to the smart device 100 or the bodythereof. For this connection, the output device 300, i.e., the body 310,may have a terminal 311, which may be inserted into the interface unit160 of the smart device 100. The head 320 is configured to emit aplurality of reference points R. Thus, the head 320 and the body 310 mayform an internal space for accommodating various components that createthe reference points R.

In terms of functionality, the output device 300 may include a lightsource 331 and a converter. The light source 331 may be configured toemit an infrared laser, in consideration of coherency and invisibility,and an infrared laser diode may be used as the light source 331. Theconverter may be configured to convert the infrared laser emitted fromthe light source 331 into the plurality of reference points R. Morespecifically, the converter may include a diffractive optical element(DOE) lens 332. The DOE lens 332 may have a plurality of microstructuresformed on the surface thereof or formed therein. The microstructures mayhave a predetermined profile and the infrared laser incident on the lens332 may be diffracted by this profile. By appropriately designing themicrostructures, the incident laser may be diffracted in a desireddirection. Furthermore, the plurality of microstructures maysimultaneously form a plurality of reference points while diffractingthe laser in a desired direction. Accordingly, the DOE lens 332 maydiffract the incident laser from the light source 331 to form aplurality of reference points emitted in desired directions.

More specifically, as shown in FIGS. 14 and 15, the head 320 may includea seat 321 formed on an outer circumferential surface thereof. The seat321 may be formed in the form of a recess, and the DOE lens 332 may beinserted into the seat 321. The head 320 may include an aperture 322formed in the bottom of the seat 321. The head 320 may have a bracket323, which is formed at the bottom of the seat 321. The bracket 323 maybe configured to accommodate the light source 331. The bracket 323extends from the bottom of the seat 321 to the inside of the head 320and may be formed to surround the light source 331. The light source 331may be supported by the bracket 323 so as to face the lens 332, and theaperture 322 may be interposed between the light source 331 and the lens332. The infrared laser emitted from the light source 331 may beincident on the DOE lens 332 through the aperture 322. The incidentlaser may be, diffracted in predetermined directions by themicrostructures in the lens 332. The diffracted laser may be convertedinto a plurality of reference points R, and the reference points R maybe uniformly distributed in the surrounding space.

Referring to FIG. 15, the output device 300 may include a substrate 334on which various electronic components for controlling the operation ofthe output device are mounted. The output device 300 may also have aswitch 336, which is mounted on the substrate 334, and the output device300 may be operated or stopped by operating the switch 336. The outputdevice 300 may be operated or stopped using the terminal 311. Control ofthe operation and stopping of the output device 300 using the switch 336and the terminal 311 has already been described in detail in relation tothe output device 200. Thus, the description of the output device 200 inrelation to operation and stopping is referred to, and furtherdescription will be omitted. The output device 300 may include a battery335 for supplying power to other components. The battery 335 may becharged by the smart device 100 or other power sources via the terminal311. Thus, the output device 300 may operate using the battery 335 evenwhen removed from the smart device 100. The components 234 to 236described above may all be accommodated in the body 310 and the head320.

Furthermore, the output device 300 may include a mechanism capable ofadjusting the orientation of the output device so as to accurately emitreference points R within the viewing angle of the user/camera 121 b. Inaddition, since the head 320 in the output device 300 emits thereference points R, the adjustment mechanism may be configured to adjustthe orientation of the head 320. Referring to FIG. 15, the output device300 may include a first flange 312, which is formed on the body 310, anda second flange 324, which is formed on the head 320 and engaged withthe first flange 312, as the adjustment mechanism. The second flange 324may be formed to at least partially surround the first flange 312 asshown in the figures. More specifically, the second flange 324 may havea bent portion, into which the first flange 312 may be inserted. Thefirst flange 312 may not be separated from the second flange 324 22 dueto such coupling, and a bearing surface may be formed between the firstflange 312 and the second flange 324. Thus, the head 320 may be rotatedabout the central axis of the output device 300 or the head 320 whilebeing guided by the bearing surface between the first flange 312 and thesecond flange 324. By this rotation, the head 320 may adjust theorientation thereof and emit the reference points R in the correctdirections. In addition, as shown in FIG. 16, the output device 300 mayhave a pair of legs 325, which are formed in the head 320, as a supportmechanism. The legs 325 may extend a long distance along the centralaxis of the head 320 and may be spaced apart from each other by apredetermined distance. Therefore, the output device 300 may be stablysupported by the legs 325 at a location remote from the smart device 100even without a separate cradle. In addition, the legs 325 may extendover body 310 to ensure more stable support.

Following the description of the output device 300, the output device400 of the modified output devices will be described in detail belowwith reference to associated drawings. FIG. 17 is a perspective viewillustrating a smart device worn by a user using a head-mounted deviceand a modified output device, and FIG. 18 is a perspective view showingthe modified output device. FIG. 19 is a perspective view showing theconversion structure of the modified output device in detail, and FIG.20 is a perspective view showing the modified output device disposed onthe floor.

Referring to FIGS. 17 and 18, in terms of structure, like the outputdevices 200 and 300, the output device 400 may include a body 410 and ahead 420 coupled to the body 410. The body 410 may be connected to thesmart device 100 or the body thereof. For this connection, the outputdevice 400, the body 410, may have a terminal 411. The terminal 411 maybe inserted into the interface 160 of the smart device 100. The head 420is configured to actually emit a plurality of reference points R. Thehead 420 and the body 410 may thus form an internal space foraccommodating various components that create the reference points R.

In terms of functionality, as shown in FIG. 19, the output device 400may include a light source 431 and a converter. The light source 431 maybe configured to emit an infrared laser, in consideration of coherencyand invisibility, and an infrared laser diode may be used as the lightsource 431. The converter may be configured to convert the infraredlaser emitted from the light source 431 into the plurality of referencepoints R. In the output device 400, the converter may include amicromirror/microscanner 432. The mirror 432 may be configured toperform two-axis rotation by an actuator. More specifically, the mirror432 may be rotated at a high speed in a tilted position by an actuatordisposed on two orthogonal axes. Thus, the mirror 432 may deflect orreflect an incident infrared laser. In addition, the mirror 432 maychange the tilt angle thereof during high-speed rotation, therebyscanning the deflected or reflected laser in desired directions. By suchreflection and scanning, the infrared laser may form a plurality ofreference points R uniformly distributed in the surrounding space.Accordingly, the mirror 432 may reflect and scan the incident laser fromthe light source 331 to form the plurality of reference points emittedin desired directions. More specifically, the head 420 may have anaperture 421 disposed adjacent to the mirror 432. The laser reflected bythe mirror 432, that is, the reference points R, may be emitted to thesurrounding space through the aperture 421.

In addition, the output device 400 may include a substrate 434 on whichvarious electronic components for controlling the operation of theoutput device are mounted. The output device 400 may be operated orstopped by manipulating the switch 436. The output device 400 may beoperated or stopped using the terminal 411. Control of the operation andstopping of the output device 400 using the switch 436 and the terminal411 has already been described in detail in relation to the outputdevice 200. Thus, the description of the output device 200 in relationto operation and stopping is referred to, and further description willbe omitted. In addition, although not shown, the output device 400 mayinclude a battery for supplying power to other components. The batterymay be charged by the smart device 100 or other power sources via theterminal 411. Thus, the output device 400 may operate using the batteryeven when removed from the smart device 100. The components 434 and 236described above may all be accommodated in the body 410 and the head420.

Furthermore, the output device 400 may include a mechanism capable ofadjusting the orientation of the output device. This adjustmentmechanism may allow the output device 400 to accurately emit referencepoints R within the viewing angle of the user/camera 121 b. In addition,since the head 410 in the output device 400 emits the reference pointsR, the adjustment mechanism may be configured to adjust the orientationof the head 410. Referring to FIG. 20, the output device 400 may includea pair of legs or flanges 422, which are formed on the head 420 androtatably connected to the body 410, as the adjustment mechanism. Morespecifically, the legs 422 may have a hinge shaft 423 formed on the bodythereof, and the hinge shaft 423 may be rotatably connected to the body410. For example, the hinge shaft 423 may be inserted into a hole formedin the body 410. Thus, the head 420 may rotate together with the legs422 around the hinge shaft 423, that is, around an axis perpendicular tothe central axis of the output device 400. By this rotation, the head420 may adjust the orientation thereof and emit the reference points Rin the correct directions. In addition, as shown in FIG. 20, the outputdevice 400 may have a bottom portion 412 of the body 410 formed to beflat as a support mechanism. The flat bottom portion 412 may allow theoutput device 400 to be stably placed on an indoor floor. Therefore, theoutput device 400 may be stably supported at a location remote from thesmart device 100 even without a separate cradle.

Since the output devices 200, 300, and 400 use infrared lasers of a highenergy level, they may be heated to unnecessarily high temperatures ifthey are used for a long time. In addition, as previously mentioned,including a number of electronic components within a small area may be acause of increasing the temperature of the output devices 200, 300 and400 while the output devices are in use. Such heating may burn the useror cause malfunction of the output device 200. Thus, the output devices200, 300, 400 may further include fins configured to dissipate heat. Asdescribed above, the output device 200 may include fins 212 as a heatdissipating structure, and fins at least functionally similar to thefins 212 may be applied to the other output devices 300 and 400. Inaddition, in terms of structure, the structure of the fins 212, whichare contractively extendable, may also be applied to the other outputdevices 300 and 400.

In order for the smart device 100 to correctly track the motion thereof,it may be advantageous that the reference points are uniformlydistributed over as wide an area as possible. However, it may be equallyimportant that such reference points R are arranged in the correctposition, i.e., in the direction of the line of sight of the camera 121b/the user. The correct arrangement of the reference points R may dependon the orientation of the output devices 200, 300 and 400. That is,orienting the output devices 200, 300, 400 in the correct direction maybe indispensable for arranging the reference points R at the correctpositions. Thus, the output devices 200, 300 and 400 may be configuredto adjust the orientations thereof. The output devices 200, 300 and 400may adjust the orientations thereof to emit the reference points Rwithin the desired space, i.e., within the viewing angle of the camera121 b/user. To enable this orientation adjustment, the output devices200, 300 and 400 may include a portion that is configured to emit thereference points R and is movable. That is, since the heads 220, 320 and420 in the output devices 200, 300 and 400 actually emit the referencepoints R, the output devices 200, 300, and 400 have the heads 210, 310and 410 movably configured for orientation adjustment. Morespecifically, the output device 200, 300, 400 may include a head 220,320, 420 that is rotatably configured for orientation adjustment. Forexample, as already described above, the head 320 of the output device300 may rotate about the central axis thereof using the first and secondflanges 312 and 324, and the head 420 of the output device 400 mayrotate about an axis perpendicular to the central axis thereof using thelegs and hinge shafts 422 and 434. These rotating structures areapplicable not only to the output device 200 but also to the outputdevices 300 and 400. Furthermore, the output device 200, 300, 400 mayinclude a portion, e.g., a head 210, 310, 420 which emits referencepoints R and is rotatable to be oriented in a plurality of directions.More specifically, the heads 210, 310 and 420 may be moved or rotated soas to be oriented toward any portion in the space with respect to theground on which the output devices 200, 300 and 400 are disposed.

Further, the reference points R are kept stationary by the outputdevices 200, 300, and 400 fixed at the remote locations. On the otherhand, the camera 121 b of the smart device 100 may move along with thedevice 100 while the smart device 100 is moving, acquire images of thereference points R at a moved point of view, and detect relative changein position from the acquired images. That is, the motion of the smartdevice 100 may be tracked by the smart device 100 itself, which actuallymoves separately from the output device at a remote location and thereference points R provided by the output device. Therefore, thereference points R emitted in the surrounding space may also be used byother smart devices. That is, the smart device 100 may share themultiple reference points R thereof with other smart devices such thatthe other devices track the motion thereof.

The smart device 100 may further include a sensing device configured tosense attachment and detachment of the output device 200, 300, 400.Various elements may be employed as the sensing device. For example, thesensing device may include a Hall sensor 238, 338, 438, which isinstalled on one of the output device 200, 300, 400 and the body of thesmart device 100, and a magnetic member, which is installed on the otherone of the output device 200, 300, 400 and the body of the smart device100. As shown in FIGS. 12, 15 and 19, the Hall sensor 238, 338, 438 maybe installed on the output device 200, 300, 400, more specifically, onthe substrate 234, 334, 434. Although not shown, the magnetic member maybe installed on the device 100. Alternatively, the Hall sensor may beinstalled on the smart device 100 and the magnetic member may beinstalled on the output device 200, 300, 400. The Hall sensors 238, 338,and 438 may sense the intensity of a magnetic field therearound. Whenthe output device 200, 300, 400 is removed from the smart device 100,the magnetic member of the smart device 100 moves away from the Hallsensor 238, 338, 438, and accordingly the sensed intensity of themagnetic field may become relatively weak. Accordingly, the sensingdevice, i.e., the Hall sensor 238, 338, 438, may sense removal of theoutput device 200, 300, 400 by sensing the intensity of the weakenedmagnetic field. On the other hand, when the output device 200, 300, 400is mounted on the smart device 100, the magnetic member of the smartdevice 100 approaches the Hall sensor 238, 338, 438. Accordingly, thesensing device, i.e., the Hall sensor 238, 338, 438, may senseattachment of the output device 200, 300, 400 by sensing the increasedintensity of the magnetic field. The operation of the output device 200,300, 400 may be controlled using the sensing of the sensing device. Theoutput device 200, 300, 400 needs to be separated from the smart device100 in order to provide fixed reference points R. That is, the outputdevice 200, 300, 400 is preferably actually operated until it is removedfrom the smart device 100. Accordingly, when the smart device 100 sensesremoval of the output device 200, 300, 400 through the sensing device,it may immediately instruct the output device 200, 300, 400 to operate.The output device 200, 300, 400 may autonomously start to operate uponsensing the removal through the sensing device. Thus, the output device200, 300, 400 may be configured to automatically operate when removed.On the other hand, the output device 200, 300, 400 may not need tooperate when mounted on the smart device 100. Accordingly, when thesmart device 100 or the output device 200, 300, 400 senses mounting ofthe device on the smart device 100 through the sensing device, theoperation of the output device 200, 300, 400 may be immediately stopped.By using the sensing device as described above, the operation of theoutput devices 200, 300, and 400 may be controlled more conveniently andeffectively.

Intended tracking of the motions of the smart device 100 and the headmay be achieved primarily by the structure of the smart device 100described above. In order to more specifically achieve such an intendedtechnical object, appropriate control needs to be supported inconsideration of the structure and characteristics of the device 100.Implementation of the functions of the smart device 100 basicallyincludes interaction with the user. Thus, through optimization ofcontrol in various aspects, including the user environment and userinterface, the above-described technical object may be achieved moreeffectively and efficiently. Furthermore, the user experience of thesmart device 100, such as ease of use and convenience, may also besignificantly improved. That is, optimum control may further increasethe commercial value of the smart device 100. For this reason, a controlmethod for the smart device 100 has been devised and will be describedhereinafter with reference to the related drawings as well as FIGS. 1 to20. FIGS. 1 to 20 and the description thereof are basically included andreferenced in the following description of the control method andrelated drawings, unless otherwise indicated.

As described above, motion tracking of the smart device 100 may berequired only in special cases. Thus, if the smart device 100 alwaystracks the motion thereof, this may waste system resources, leading toshortening of operation time and malfunction. Therefore, it may beimportant to determine a time to start tracking motion in providing thebest virtual reality in the smart device 100. Therefore, the controlmethod described below may include determining the time. The controlmethod may also include a step of preparing the smart device 100 and theoutput device 200, 300, 400 thereof to track the motion after thetracking is determined. FIG. 21 is a flowchart illustrating an exampleof a method for controlling a smart device described in the presentapplication. FIG. 22 is a conceptual diagram showing examples ofnotifications provided to the user in the method for controlling a smartdevice. As mentioned above in describing the structure of the smartdevice 100, the control unit 180 may be referred to by various namessuch as a controller and a control device and may control all componentsof the smart watch 100 in order to perform a predetermined operation.Thus, the control unit 180 may substantially control the method includedin this application, namely, all the steps according to FIGS. 8, 21 and22, and therefore all the steps which will be described hereinafter maybe features of the control unit 180.

First, the smart device 100 may check if it needs to track the motionthereof (S10). The smart device 100 does not need to always track themotion thereof, and may be required to track the motion thereof only inspecial cases. First, the smart device may detect whether a specificapplication is executed (S11). The specific application may be, forexample, a gaming application that is manipulated by the motion of theuser's head. The specific application may be any application thatrequires tracking of the motion of the user's head. When the specificapplication is executed, the smart device 100 may determine thattracking the motion thereof is needed. That is, the smart device 100 maydetermine to start tracking the motion according to such necessity. Inaddition, for faster determination, the smart device 100 may detectwhether a preliminary operation for executing the specific applicationis performed prior to actual execution of the specific application. Thispreliminary operation may be, for example, an operation of establishingthe settings before executing a game. Once the preliminary operation isperformed, the smart device 100 may determine tracking of the motion.

The sensors in the smart device 100, for example, gyro sensors oracceleration sensors, may continuously detect and track the rotationalmotion, but cannot continuously detect and track the translationalmotion, which constitutes most of the motion of the head. However, sincethese sensors may detect translational motion within a limited range (ordegree), it may be possible to detect whether such translational motionshave occurred, even though the actual degree of translational motionsoccurring beyond a relatively limited range may not be detected.Accordingly, the smart device 100 may detect whether the motion of thesmart device 100, particularly the translational motion, occurs beyond acertain range (S12). If the motion of the smart device 100 occurs beyondthe certain range, the smart device 100 may determine that motion of thehead has substantially occurred. Further, based on this determination,the smart device 100 may determine that the function it is providingsubstantially requires tracking of the motion of the smart device 100.Accordingly, the detection step S12 may be performed when the smartdevice 100 provides a predetermined function, for example, when thesmart device 100 performs a predetermined operation or executes apredetermined application. As a result, if the motion of the smartdevice 100 occurs beyond the certain range, the smart device 100 maydetermine that it needs to track the motion thereof and may accordinglydetermine to start motion tracking.

When it is determined that the motion of the smart device 100 needs tobe tracked in the checking step S10, that is, when such tracking of themotion is substantially determined, the smart device 100 may search fora plurality of reference points (R) present in a surrounding space(S20). In the searching step S20, the smart device 100 may acquire animage of the surrounding space using the camera 121 b. Subsequently, thesmart device 100 may analyze the acquired image to check if there is aplurality of reference points (R). When the reference points R areformed by infrared light and the camera 121 b is configured as aninfrared camera, the camera 121 b may acquire images of infraredobjects, which are invisible. Therefore, it may be easily checkedwhether there are reference points R in the surrounding space bychecking the infrared camera 121 b for the reference points R having apredetermined shape. In addition, as described above, the referencepoints R may remain fixed and motion tracking of the smart device 100may be performed by the smart device 100, which actually moves,separately from the provision of the reference points R. Accordingly,the reference points R emitted in the surrounding space may be shared bydifferent smart devices 100. Thus, in the searching step S20, thereference points R emitted not only from the output device 200, 300,400, but also from output devices of others devices may be searched for.For simplicity, the following description refers to only the outputdevice 200 among the output devices, but it will be understood that eachof the steps described below includes both the use of the output device200 and use of other output devices 300, 400.

When the smart device 100 finds the reference points R in thesurrounding space in the searching step S20, the smart device 100 mayimmediately start motion tracking. In this tracking process, since thereference points R are already provided in the surrounding space, it maynot be necessary to additionally provide reference points R in thesurrounding space. Therefore, except for the emitting step S1 of FIG. 5,steps S3 to S5 may be sequentially performed to track the motion,starting with the acquiring step S2, as indicated by a symbol “A” inFIGS. 5 and 8.

If the smart device 100 fails to find the reference points R in thesurrounding space in the searching step S20, a request for the referencepoints R may be made to the smart device 100 first in order to track themotion. Accordingly, in order to secure the reference points R, thesmart device 100 may additionally search for an available output device200 (S30). The output device 200 may be available regardless of whetherit is mounted on or removed from the smart device 100. That is, themounted output device 200 may be removed from the smart device 100 foroperation. If the output device 200 is already removed, it may beoperated immediately to emit the reference points R. Accordingly, in thesearching step S30, the smart device 100 may search for both the outputdevice 200 mounted on the smart device 100 and the output device 200removed from the smart device 100. This searching step may be performedusing a sensing device as described above. For example, the mounted orremoved output device 200 may be searched for by the Hall sensor and themagnetic member installed on the output device 200 and the device 100,respectively. Further, when the output device 200 is additionallyconfigured to perform communication, the smart device 100 may easilysearch for the output device 200 removed therefrom. For example, theoutput device 200, which may include various communication modules, mayinclude a near field communication module, for example, an NFC tag, asthe simplest form of communication module. Therefore, even if removedfrom the smart device, the output device 200 including such acommunication module may be easily searched for by the smart device 100.Further, as described above, the reference points R may be shared bydifferent smart devices 100. Accordingly, in the searching step S30, thesmart device 100 may search for both the output device of the smartdevice 100 and the available output devices of the other smart devices.Similarly, if the output device 200 of another smart device includes acommunication module such as an NFC tag, it may be easily searched forby the smart device 100. Further, since the output devices mounted on orremoved from the smart device 100 and other smart devices are allavailable devices as described above, the output devices 200 may besearched for by the smart device 100 in the searching step S30.

If the smart device 100 fails to find any available output device 200 inthe searching step S30, the smart device 100 may notify the user that anavailable output device 200 should be searched for (S31). For example,as shown in FIG. 22(a), the smart device 100 may provide, on the displayunit 151, a notification such as “Please find an available output devicefor motion tracking”. After providing the notification, the smart device100 may temporarily suspend the function currently being provided. Thatis, since the function using the motion tracking cannot be provided, theoperation or the application that is being executed may be temporarilystopped. At the same time, the smart device 100 may continuously searchfor an available output device 200 in the searching step S30. Inaddition, since the user may bring the output device 200 and immediatelyoperate the same to provide the reference points R, existence of thereference points may also be continuously searched for in the searchingstep S20.

When the smart device 100 finds an available output device 200 in thesearching step S30, the smart device 100 may operate the availableoutput device 200 (S40). Fixed reference points R should be provided todetect change in position. Thus, the output device 200 needs to beprimarily removed or detached from the smart device 100 so as not tomove along with the smart device 100. Therefore, if the output device200 is mounted on the smart device 100 in the operating step S40, theoutput device 200 may first be removed or detached from the smart device100. In addition, to provide uniform reference points R, the outputdevice 200 needs to be stably arranged at a remote location away fromthe smart device 100 after being removed and detached. As describedabove, for this arrangement, the output device 200 may have a supportstructure, and a separate support structure such as a cradle may also beused. If the output device 200 is already removed from the smart device100, then the output device 200 may be stably arranged in a similarmanner for emission of reference points R. The output device 200 ofanother smart device 100 operable due to the shareable reference pointsR may also be operatively prepared according to the same procedure asdescribed above. That is, the other mounted output device 200 may beremoved from the other smart device 100, and be stably arranged.

As described above, the actual operation of the output device 200requires removal of the output device 200 first. Accordingly, the smartdevice 100 may notify the user that the output device 200 should beremoved from the smart device 100 before the operating step S40 isperformed (S42). For example, as shown in FIG. 22(b), the smart device100 may provide, on the display unit 151, a notification such as “Pleaseremove the attached output device and arrange it at a proper place toperform motion tracking”. More specifically, the notifying step S42 maybe performed before the output device 200 is actually removed from thesmart device 100 in the operating step S40 because there is an availableoutput device 200. Furthermore, since the notification corresponds to apreliminary or precautionary alert, the notifying step S42 may beperformed to provide the notification, regardless of whether or not theavailable output device 200 is mounted on or removed from the smartdevice 100. In the notifying step S42, the user may be pre-guided by thenotifying step S42 to perform the operating step S40. Accordingly, theuser may use the smart device 100 more conveniently, and motion trackingof the smart device 100 may be performed more reliably.

In addition, before the notifying step S42, the smart device 100 mayadditionally sense attachment or detachment of the output device 200(S41). As already described above, attachment or detachment of theoutput device 200 may be sensed using a sensing device or a short-rangecommunication module. Since the output device 200 is electricallyconnected to the smart device 100 using the terminal 211, presence orabsence of the electrical connection may allow the smart device 100 toeasily sense the attachment or detachment of the output device 200.Further, when communicatively connected to another smart device, thesmart device 100 may receive attachment or detachment of the outputdevice of the other smart device from the other smart device. If it issensed that the output device 200 is mounted on the other smart device,the smart device 100 may notify the user of detachment of the outputdevice 200 through the notifying step S42. If it is sensed that theoutput device 200 is removed, the, the smart device 100 may notify theuser that the removed output device 200 should be operated. The smartdevice 100 may directly instruct the removed output device 200 equippedwith the communication module to operate. Furthermore, the output device200 may be configured to operate automatically when removed from thesmart device. In this case, the smart device 100 may instruct the outputdevice 200 to operate upon sensing detachment of the output device 200,or the output device 200 may automatically start to operate when removedfrom the smart device.

As described above, when the removed output device 200 is disposed at anappropriate remote location, the output device 200 may be operated toactually emit reference points. For example, the output device 200 maybe operated by pressing the switch 236. In addition, if the outputdevice 200 includes a communication module, the smart device 200 mayremotely support the operation of the output device 200. Thus, steps S3to S5 may be sequentially performed to track the motion, starting withthe emitting step S1, as indicated by a symbol “B” in FIGS. 5 and 8.

In the control method of the present application described above, thetime of occurrence of motion may be determined by detecting a specialcase in which motion tracking is needed. In addition, after tracking isdetermined, the control method may appropriately prepare the smartdevice 100 and the output device 200, 300, 400 thereof such that motionmay be tracked. That is, in the control method, the previously providedreference points R and the available output devices 200, 300, and 400may be searched for, and a notification for additional search andoperation of the available output devices 200, 300, and 400 may beprovided to the user at an appropriate time. Thus, the control methodmay effectively track motion of the smart device 100 and the head, andallow the user to more easily utilize virtual reality.

Therefore, the above embodiments should be construed in all aspects asillustrative and not restrictive. The scope of the disclosure should bedetermined by the appended claims and their equivalents, and all changescoming within the meaning and equivalency range of the appended claimsare intended to be embraced therein.

1.-30. (canceled)
 31. A smart device comprising: a body; an outputdevice detachable from the body and configured to provide a plurality ofreference points in a surrounding space; a camera configured to acquireimages of the provided reference points; and a control device configuredto track motion of the smart device using the acquired images.
 32. Thesmart device according to claim 31, wherein the output device and thecamera continuously provide reference points and acquire images thereofwhile the smart device is moving, wherein the reference points havepredetermined patterns to be easily recognized, the patterns beingdifferent from each other, and wherein the output device providesreference points formed by infrared light, and the camera comprises aninfrared camera capable of photographing the infrared reference points.33. The smart device according to claim 31, wherein the control deviceis configured to: construct a three-dimensional map of predeterminedreference points based on positions of a predetermined number of thepredetermined reference points acquired from the images, continuouslydetect a relative positional change of the predetermined referencepoints with respect to the three-dimensional map using the images, andcontinuously detect a position of the smart device based on the detectedpositional change, and extend the three-dimensional map based onpositions of additional reference points as a range of motion of thesmart device extends.
 34. The smart device according to claim 31,wherein the output device comprises: a light source configured to emitinfrared laser; and a converter configured to convert the emittedinfrared laser into the plurality of reference points.
 35. The smartdevice according to claim 34, wherein the converter comprises adiffusing member configured to uniformly disperse the emitted infraredlaser, and wherein the converter further comprises a filter configuredto selectively pass the infrared laser dispersed by the diffusing memberto emit the reference points.
 36. The smart device according to claim34, wherein the converter is configured to diffract the emitted infraredlaser to form the plurality of reference points, or wherein theconverter is configured to reflect the emitted infrared laser in aplurality of directions to form the plurality of reference points. 37.The smart device according to claim 31, wherein the output device isdisposed at a location spaced apart from the body to provide theplurality of reference points, wherein the output device furthercomprises a fin configured to dissipate heat and to contractively extendfrom the output device, wherein the output device is configured toadjust an orientation thereof to provide the reference points in adesired space, and wherein the smart device shares a plurality ofreference points provided by an output device of another smart device totrack the motion thereof.
 38. The smart device according to claim 31,further comprising a head-mounted device configured to accommodate thebody, the head-mounted device being worn on a head of a user, whereinthe controller tracks motions of the head-mounted device and the head ofthe user wearing the smart device by tracking the motion of the smartdevice, and further comprising a sensing device configured to senseattachment and detachment of the output device.
 39. A method forcontrolling a smart device, the method comprising: searching for aplurality of reference points in a surrounding space when it isnecessary to track motion of a smart device; emitting the referencepoints and searching for an output device available in the smart devicewhen the plurality of reference points is not found; and operating theavailable output device when the output device is found.
 40. The methodaccording to claim 39, wherein the operating comprises: removing anoutput device mounted on the smart device from the smart device; anddisposing the removed output device at a position spaced apart the smartdevice.
 41. The method according to claim 39, further comprisingnotifying a user that the output device should be removed from the smartdevice so as to be used at a remote location, the notifying beingperformed prior to the operating step, and further comprising sensingwhether the output device has been removed from the smart device, thesensing being performed prior to the notifying.
 42. The method accordingto claim 39, wherein the operating comprises: operating one of theoutput device of the smart device and an output device of another smartdevice.
 43. The method according to claim 39, further comprising:notifying a user that the available output device should be searched forwhen the available output device is not found.
 44. The method accordingto claim 39, wherein the searching is performed when the smart deviceexecutes a specific application or when motion of the smart deviceoccurs beyond a predetermined range.
 45. The method according to claim39, further comprising, when the plurality of reference points is foundin the searching: continuously acquiring, by the smart device, images ofthe reference points while the smart device is moving; constructing athree-dimensional map of predetermined reference points based onpositions of a predetermined number of the predetermined referencepoints acquired from the images; continuously detecting a relativepositional change of the predetermined reference points with respect tothe constructed three-dimensional map using the images; and continuouslydetecting a position of the smart device based on the detectedpositional change.
 46. The method according to claim 39, wherein thesearching of the output device comprises: searching for an output devicemounted on the smart device and an output device removed from the smartdevice; or searching for an output device of the smart device and anavailable output device of another smart device.
 47. A smart devicecomprising: a body; an output device detachable from the body andconfigured to provide a plurality of reference points in a surroundingspace; a camera configured to acquire images of the provided referencepoints; and a control device configured to track motion of the smartdevice using the acquired images, wherein, when the plurality ofreference points is not found through the camera, the control devicenotifies a user that the output device should be removed from the smartdevice before the output device is operated.
 48. The smart deviceaccording to claim 47, wherein the control device immediately operatesthe output device when it is sensed that the output device has beenremoved, wherein, when it is sensed that the output device is mounted,the control device notifies the user that the output device should beremoved.
 49. The smart device according to claim 47, wherein the controldevice is configured to share reference points provided by an outputdevice of another smart device to track the motion of the smart device.50. The smart device according to claim 47, wherein, when the pluralityof reference points is not found, the control device is configured tooperate one of the output device of the smart device and an outputdevice of another smart device, and wherein, when there is no availableoutput device, the control device notifies the user that an availableoutput device should be searched for.